Skip to content

NLTK Removal #1102

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 24 commits into from
Oct 21, 2024
Merged

NLTK Removal #1102

merged 24 commits into from
Oct 21, 2024

Conversation

@AlejandroEsquivel
Copy link
Member

@AlejandroEsquivel AlejandroEsquivel commented Sep 27, 2024 •
edited
Loading

Updated:

  • Removed nltk in favour of a rule based sentence splitter
  • Ported and modified WordTokenizers.jl sentence splitter, namely to account for more abbreviations, not splitting in between quotations, using custom separator as opposed to new line character which could erroneously split new line characters in original text that aren't the end of a sentence

Removed Packages:

  • nltk
  • griffe
  • tenacity

@AlejandroEsquivel
removed nltk dependency
bbef78a
@AlejandroEsquivel
remove nltk import and download from validator base
0298730
@AlejandroEsquivel
remove commented out test referencing nltk
e699397
@AlejandroEsquivel
throwing import error if nltk not available in detect_pii mock
c1f7350
@AlejandroEsquivel
Added rule based sentence tokenization from WordTokenizers.jl with mi…
416f290 
…nor modifications
@AlejandroEsquivel
Added new version of split_sentence_str using new rule based sentence…
6743df2 
… tokenization
@AlejandroEsquivel
updates to factor in quotes during sentence splitting
acae737
@AlejandroEsquivel
updated poetry.lock
815300f
@AlejandroEsquivel
replaced split sentence default
1a2546a
@AlejandroEsquivel
testing changes using custom separator in wordtokenizer algo
bd0ab50
@AlejandroEsquivel
fix for counting subs
4ebef18
@AlejandroEsquivel
reverted split sentence in validators base
fb1d2d0
@AlejandroEsquivel
reverted to pre-seperator algo, added fix for conditional white space…
3fb5765 
… after ?!. chars
@AlejandroEsquivel
fix for optional white space after potential line endings ?!.
f453ecc
@AlejandroEsquivel
Merge branch 'main' into feat/nltk-removal
88477c7
@AlejandroEsquivel
added back modified seperator algo, fix for split sentence
6462558
@AlejandroEsquivel
Fix regex patterns for abbreviations in tokenization_utils_seperator.py
6f8b175
@AlejandroEsquivel
fix tests
27a0419
@AlejandroEsquivel
remove nltk references from tests
49aa02d
@AlejandroEsquivel
removed older scripts
affb205
@AlejandroEsquivel AlejandroEsquivel marked this pull request as ready for review October 1, 2024 21:51
@zsimjee zsimjee added this to the v0.6.0 release tracker milestone Oct 7, 2024
@zsimjee zsimjee changed the base branch from main to 0.6.0-dev October 18, 2024 21:05
@zsimjee zsimjee mentioned this pull request Oct 21, 2024
Closed
@dtam dtam merged commit 8ceb5db into 0.6.0-dev Oct 21, 2024
12 checks passed
@dtam dtam deleted the feat/nltk-removal branch October 21, 2024 21:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zsimjee zsimjee zsimjee approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

v0.6.0 release tracker

Development

Successfully merging this pull request may close these issues.

4 participants

@AlejandroEsquivel @zsimjee @dtam